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Preface  &  Acknowledgements 


During  his  internship  with  the  Graduate  School  of  Business  &  Public  Policy  in  June 
2010,  U.S.  Air  Force  Academy  Cadet  Chase  Lane  surveyed  the  activities  of  the  Naval 
Postgraduate  School’s  Acquisition  Research  Program  in  its  first  seven  years.  The  sheer 
volume  of  research  products — almost  600  published  papers  (e.g.,  technical  reports,  journal 
articles,  theses) — indicates  the  extent  to  which  the  depth  and  breadth  of  acquisition 
research  has  increased  during  these  years.  Over  300  authors  contributed  to  these  works, 
which  means  that  the  pool  of  those  who  have  had  significant  intellectual  engagement  with 
acquisition  issues  has  increased  substantially.  The  broad  range  of  research  topics  includes 
acquisition  reform,  defense  industry,  fielding,  contracting,  interoperability,  organizational 
behavior,  risk  management,  cost  estimating,  and  many  others.  Approaches  range  from 
conceptual  and  exploratory  studies  to  develop  propositions  about  various  aspects  of 
acquisition,  to  applied  and  statistical  analyses  to  test  specific  hypotheses.  Methodologies 
include  case  studies,  modeling,  surveys,  and  experiments.  On  the  whole,  such  findings 
make  us  both  grateful  for  the  ARP’s  progress  to  date,  and  hopeful  that  this  progress  in 
research  will  lead  to  substantive  improvements  in  the  DoD’s  acquisition  outcomes. 

As  pragmatists,  we  of  course  recognize  that  such  change  can  only  occur  to  the 
extent  that  the  potential  knowledge  wrapped  up  in  these  products  is  put  to  use  and  tested  to 
determine  its  value.  We  take  seriously  the  pernicious  effects  of  the  so-called  “theory- 
practice”  gap,  which  would  separate  the  acquisition  scholar  from  the  acquisition  practitioner, 
and  relegate  the  scholar’s  work  to  mere  academic  “shelfware.”  Some  design  features  of  our 
program  that  we  believe  help  avoid  these  effects  include  the  following:  connecting 
researchers  with  practitioners  on  specific  projects;  requiring  researchers  to  brief  sponsors  on 
project  findings  as  a  condition  of  funding  award;  “pushing”  potentially  high-impact  research 
reports  (e.g.,  via  overnight  shipping)  to  selected  practitioners  and  policy-makers;  and  most 
notably,  sponsoring  this  symposium,  which  we  craft  intentionally  as  an  opportunity  for 
fruitful,  lasting  connections  between  scholars  and  practitioners. 

A  former  Defense  Acquisition  Executive,  responding  to  a  comment  that  academic 
research  was  not  generally  useful  in  acquisition  practice,  opined,  “That’s  not  their  [the 
academics’]  problem — it’s  ours  [the  practitioners’].  They  can  only  perform  research;  it’s  up 
to  us  to  use  it.”  While  we  certainly  agree  with  this  sentiment,  we  also  recognize  that  any 
research,  however  theoretical,  must  point  to  some  termination  in  action;  academics  have  a 
responsibility  to  make  their  work  intelligible  to  practitioners.  Thus  we  continue  to  seek 
projects  that  both  comport  with  solid  standards  of  scholarship,  and  address  relevant 
acquisition  issues.  These  years  of  experience  have  shown  us  the  difficulty  in  attempting  to 
balance  these  two  objectives,  but  we  are  convinced  that  the  attempt  is  absolutely  essential  if 
any  real  improvement  is  to  be  realized. 
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whose  foresight  and  vision  have  assured  the  continuing  success  of  the  Acquisition 
Research  Program: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics) 

•  Program  Executive  Officer  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Program  Manager,  Airborne,  Maritime  and  Fixed  Station  Joint  Tactical  Radio  System 
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Science  and  Technology,  Enterprise  Systems  and  Planning,  and  Operations  Analysis  and 
Management  mission  areas.  He  is  responsible  for  developing  and  delivering  services  that  enable  the 
clients  of  Analytical  Services,  Inc.,  to  address  critical  challenges  in  national  security  and  public  safety, 
and  to  improve  the  effectiveness  of  public-sector  programs.  Dr.  McGrath  leads  a  workforce  whose 
expertise  spans  a  wide  range  of  technology  and  application  domains  in  research,  acquisition, 
information  systems  and  defense  operations. 


Dr.  McGrath  served  as  Deputy  Assistant  Secretary  of  the  Navy  for  Research,  Development,  Test,  and 
Evaluation  from  February  2003  to  September  2007.  His  role  was  to  aggressively  drive  new 
technologies  from  all  sources  across  Navy  and  Marine  Corps  platforms  and  systems  and  to  develop 
programs  to  bridge  the  gap  in  transitioning  from  science  and  technology  to  acquisition.  He  was  also 
responsible  for  integrating  test  and  evaluation  with  the  evolutionary  acquisition  process.  His 
leadership  was  key  to  the  restructuring  of  the  Future  Naval  Capabilities  program,  the  success  of  the 
Rapid  Technology  Transition  program,  and  the  establishment  of  the  Navy  Enterprise  T&E  Board  of 
Directors  and  the  Navy  Lab  and  Centers  Competency  Group. 

Prior  to  his  return  to  government  service  in  2003,  Dr.  McGrath  spent  five  years  as  vice  president  for 
Government  Business  at  the  Sarnoff  Corporation,  a  leading  R&D  company  with  both  commercial  and 
government  clients.  He  was  responsible  for  developing  programs  to  meet  government  needs  for 
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Dr.  McGrath’s  previous  government  experience  includes  weapon  system  logistics  planning  and 
management  at  Naval  Air  Systems  Command,  acquisition  policy  in  the  Office  of  the  Secretary  of 
Defense,  and  several  technology  management  positions.  He  was  the  first  OSD  Director  of  the 
Computer-aided  Acquisition  and  Logistics  Support  program.  At  DARPA,  he  managed  programs  in 
Agile  Manufacturing,  Electronic  Commerce  Resource  Centers,  and  Affordable  Multi  Missile 
Manufacturing.  He  also  served  in  leadership  positions  for  several  DoD-wide  initiatives  to  improve 
manufacturing  and  reduce  the  cost  of  defense  systems.  As  the  Assistant  Deputy  Under  Secretary  of 
Defense  (Dual  Use  and  Commercial  Programs),  he  directed  the  Commercial  Technology  Insertion 
Program,  the  Commercial  Operating  and  Support  Savings  Initiative,  and  the  Department’s  Title  III 
industrial  base  investments. 
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Dr.  McGrath  holds  a  BS  in  Space  Science  and  Applied  Physics  (1970)  and  an  MS  in  Aerospace 
Engineering  (1972)  from  Catholic  University,  and  a  doctorate  in  Operations  Research  from  George 
Washington  University  (1985).  He  was  an  adjunct  associate  professor  at  GWU  in  1987-1988.  He  is 
active  in  several  industry  associations  and  study  groups,  including  studies  by  the  Defense  Science 
Board  and  the  National  Research  Council. 
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Making  Acquisition  Measurable 

Kevin  Buck — Principal  Economics  and  Business  Analyst,  Center  for  Acquisition  and  Systems 
Analysis  (CASA).  Mr.  Buck  provides  investment,  portfolio,  and  performance  analyses  and 
management  solutions  for  Defense,  Intelligence,  and  Civilian  Agency  sponsors.  He  is  a  Principal 
Investigator  for  a  MITRE  research  project  related  to  streamlining  transparency,  accountability,  and 
performance  management  Improvement.  Mr.  Buck  has  a  Bachelor  of  Science  in  Marine 
Transportation  from  the  U.S.  Merchant  Marine  Academy,  and  a  Master  of  Science  in  Industrial 
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Engineering  (Oklahoma  State),  Wire  Communications  Technology,  and  Business  Administration 
(Wayland  Baptist)  and  a  Master  of  Science  degree  in  Systems  Engineering  from  Johns  Hopkins 
University,  [dhanf@mitre.org] 


Abstract 


The  ultimate  objective  of  our  investigations  was  to  establish  a  foundation  for 
improving  how  acquisition  performance  is  managed.  Our  project  looked  broadly 
across  the  four  principles  highlighted  in  NDAA  Section  804  and  subsequently 
focused  on  the  challenges  that  program  managers  might  face  in  measuring  adoption 
and  impact  of  the  user  engagement  principle.  We  discovered  that  the  principles  are 
interrelated  and  that  an  understanding  of  how  acquisition  success  will  be  measured 
is  critical  to  understanding  the  principles’  contribution  to  successful  acquisition 
outcomes.  Our  research  focused  most  extensively  on  the  challenges  that 
Government  program  offices  face  in  ensuring  early  and  continual  involvement  of  the 
user,  measuring/monitoring  user  engagement  in  achieving  program/system 
objectives,  and  determining  the  impact  of  user  involvement.  Based  on  direct 
interaction  with  users  of  Government  systems  and  program  capabilities,  our  research 
resulted  in  the  identification  of  essential  elements  for  an  effective  user  engagement 
program,  codification  of  key  user  types  and  characteristics,  candidate  high-priority 
user  engagement  metrics,  lessons  learned  in  deriving  metrics,  relevance  of 
performance  management  principles  for  measuring  user  engagement,  and  insights 
from  users  for  improving  how  program  offices  can  more  effectively  and  efficiently 
engage  users  in  the  process  of  delivering  required  capabilities. 

Report  Summary 


“Information  technology  (IT)  offers  immense  capability  in  terms  of  agility,  flexibility, 


responsiveness,  and  effectiveness.  It  enables  nearly  all  of  our  military  combat  capability  and 
has  become  a  necessary  element  of  our  most  critical  warfare  systems.  However,  there  is 
growing  concern  within  Congress  and  among  DOD  leadership  that  the  nation’s  military 
advantage  may  be  eroding.  The  deliberate  process  through  which  weapon  systems  and 
information  technology  are  acquired  by  DOD  cannot  keep  pace  with  the  speed  at  which  new 
capabilities  are  being  introduced  in  today’s  information  age — and  the  speed  with  which 
potential  adversaries  can  procure,  adapt,  and  employ  those  same  capabilities  against  the 


United  States. 
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-  Defense  Science  Board  (2009) 


MITRE  Technical  Report  (MTR)  110102,  “Making  Acquisition  Measurable — FY  2010 
NDAA  Section  804  Principles”  summarizes  the  results  of  preliminary  investigations 
undertaken  by  MITRE’s  “Making  Acquisition  Measurable”  (MAM)  Capability  Development 
Team  to  support  Government  programs  in  measuring  the  adoption  and  impact  of  four 
Information  Technology  (IT)  acquisition  principles.  These  principles  are  identified  within 
Section  804  of  the  FY  2010  National  Defense  Authorization  Act  (NDAA)  as  critical  to  a  new 
IT  acquisition  process  that  must  be  created  by  the  Department  of  Defense  (DoD).  The  four 
principles  include: 

■  Early  and  continual  involvement  of  the  user; 

■  Multiple,  rapidly  executed  increments  or  releases  of  capability; 

■  Early,  successive  prototyping  to  support  an  evolutionary  approach;  and 

■  A  modular,  open-systems  approach. 

The  ultimate  objective  of  our  investigations  was  to  establish  a  foundation  for 
improving  how  acquisition  performance  is  managed.  The  absence  of  a  formalized  and 
standard  performance  management  methodology  has  been  noted  by  the  Flouse  Armed 
Services  Committee  Panel  on  Defense  Acquisition  Reform  as  a  critical  area  of  weakness. 

To  create  a  foundation  for  improved  performance  management,  the  team  needed  to  better 
understand  how  program  managers  can  more  effectively  and  efficiently  do  the  following: 

■  Account  for  the  unique  nature  of  IT  in  their  performance  measurement  and 
program  management; 

■  Apply  performance  metrics  to  determine  whether  desired  outcomes  from  their 
programs  and  acquisitions  will  likely  be  achieved; 

■  Identify  in  a  proactive  manner  whether  course  corrections  are  needed  or 
expectations  should  be  adjusted; 

■  Leverage  best  practices,  lessons  learned,  and  existing  tools/analyses  to 
improve  data  collection,  performance  measurement,  acquisition  monitoring, 
and  acquisition  execution  decision-making;  and 

■  Ensure  that  performance  management  efforts  support  improved  performance 
(e.g.,  timely  delivery  of  required  capabilities,  services,  or  products  to  the  end- 
user). 

The  project  looked  broadly  across  the  four  principles  highlighted  in  NDAA  Section 
804  and  subsequently  focused  on  the  challenges  that  program  managers  might  face  in 
measuring  adoption  and  impact  of  the  user  engagement  principle.  We  discovered  that  the 
principles  are  interrelated  and  that  an  understanding  of  how  acquisition  success  will  be 
measured  is  critical  to  understanding  the  principles’  contribution  to  successful  acquisition 
outcomes.  Figure  1  illustrates  key  relationships  among  the  four  principles. 


NP SI 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  - 133 


Prototypes  demonstrate 
feasibility  to  users  and 
providea  tangible  means 
to  get  more  definition  on 
users  needs 


Early,  Successive 
Prototyping  to  Support  an 
Evolutionary  Approach 


MOSA  enables  insertion  of 
and  demonstration  of 
prototypes  in  an 
operationally  releva nt 
environment 


MOSA  makes  the 
system  more  flexible 
and  thus  easierto 
accommodate 
changed  user  needs 


User  provides  priorities  for 
increments.  Increments 
provide  opportunity  for 
frequent  feedback. 


Prototypes  provide  a 

-risk  reduction - > 

method  for  defining 
increment  content. 
Prototypes  can  be 
developed 
incrementally  to 
becomethe  final 
delivery  < 


A  Modular,  Open  Systems 
Approach  (MOSA) 


Multiple,  Rapidly 
Executed  Increments  or 
Releases  of  Capability 


A  stable  MOSA-based 
environment  is  the 
foundationforan 
incremental  delivery  culture 


Figure  1.  Relationship  Among  Section  804  Principles 

Incremental  deliveries  and  supporting  underlying  processes,  for  instance,  should 
provide  early  opportunities  to  learn  about  the  changing  user  environment.  Designing  user 
interfaces  using  MOSA  principles  should  reduce  user  training  needs  and  allow  for  more 
rapid  deployment  of  components  that  have  changed  because  of  user  needs.  Applying 
prototypes  should  provide  a  tangible  presentation  of  the  evolving  or  proposed  system,  which 
should  enrich  user-developer  interactions. 

Our  research  focused  most  extensively  on  the  challenges  that  Government  program 
offices  face  in  ensuring  early  and  continual  involvement  of  the  user,  measuring/monitoring 
user  engagement  in  achieving  program/system  objectives,  and  determining  the  impact  of 
user  involvement.  Based  on  direct  interaction  with  users  of  Government  systems  and 
program  capabilities,  our  research  resulted  in  the  identification  of  essential  elements  for  an 
effective  user  engagement  program,  codification  of  key  user  types  and  characteristics, 
candidate  high-priority  user  engagement  metrics,  lessons  learned  in  deriving  metrics, 
relevance  of  performance  management  principles  for  measuring  user  engagement,  and 
insights  from  users  for  improving  how  program  offices  can  more  effectively  and  efficiently 
engage  users  in  the  process  of  delivering  required  capabilities. 
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Motivation 


•  Department  of  Defense  (DoD)  was  directed  by  Congress  to 
design  a  new  IT  acquisition  process 

o  Direction  references  Chapter  6  of  the  March  2009  Defense  Science  Board 
(DSB)  Task  Force  Report  on  Policies  and  Procedures  for  the  Acquisition  of  IT 

o  The  mandate  targets  four  principles: 

*  Early  and  continual  user  involvement 

Multiple,  rapidly  executed  increments  or  releases  of  capability 

*  Early,  successive  prototyping  to  support  an  evolutionary  acquisition 

*  Modular,  open  systems  approach  (MOSA) 

•  How  will  programs  measure,  monitor,  and  report  adoption  of 
the  principles  in  the  new  process? 

o  Need  a  foundation  for  improving  how  acquisition  performance  is  managed 

According  to  the  House  Armed  Services  Committee  Panel  on  Defense  Acquisition 
Reform,  a  critical  area  of  weakness  is  the  lack  of  a  formalized  performance 
management  methodology 


*2010  National  Defense  Authorization  Act  (NDAA)  Section  804,  "Implementation  of  New  Acquisition  Process  for  IT  Systems" 


MITRE 


©201 1  The  MITRE  Corporation.  All  rights  reserved 


3 


Command  Si  Control  Center 


Objective 

Help  programs  effectively  measure,  monitor,  and 
report  progress  in  achieving  IT  acquisition  outcomes 


o  Desired  impact: 

Increased  agility 
*  Reduced  cost  growth 


HIM 

1 1 

W  ^  ^ 

um 

Flexibility 


x  Increased  meaningful  deliveries 


Diagnose 


Improvement 

Readiness 


Improvement 


Improvement  Goals 


Enablers 

and 

Constraints 


Process  and/or  Management  Maturity 


Tailor  L 


Recommend 


Propose  a  diagnostic  to  help  IT 
programs  manage  performance 

Accommodate  tailoring 
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Engaging  with  Users 


•  Our  focus  was  on  Early  and  Continual  User  Involvement  (Ul) 

o  We  first  interviewed  users  across  Federal  government  programs 

•  What  we  heard  from  users  during  our  interviews: 

o  "We  liked  it  when  they  came  to  us,  showed  us  a  new  capability  and  then 
returned  with  changes  that  we  had  suggested" 

o  Developers  should  not  be  involved  too  early  in  the  process 

o  "The  program  office  should  come  out  and  see  the  pain  that  we  experience 
using  the  system;  they  would  understand  the  requirement  better" 

o  "Users  should  also  talk  amongst  themselves" 

o  User  representatives  in  the  program  office  should  come  from  the  users' 
organization 

o  "Consistency  in  interactions  on  a  cadence  that  is  predictable  is  important  to 
obtaining  desired  capabilities" 


MITRE 


©201 1  The  MITRE  Corporation.  All  rights  reserved 


5 


Command  &  Control  Center 


Types  of  Users  Targeted 

We  identified  a  number  of  different  types  of  users  with 
whom  Program  Offices  typically  must  interact 


User  Advocate  (UA) 
User  Representative 


Subject  Matter 
Expert  (SME) 


End  Users 


Administrator/ 
Maintainer  (A/M) 


Manager,  Evaluator,  Supervisor,  Task  Performer 
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Proposed  User  Engagement  Program 


•  Based  on  our  investigations,  we  recommend  key 
elements  of  a  "User  Engagement  Program" 


Effective  and  Efficient 
User  Engagement 


Fundamental  Leading  Indicators: 


(1)  Are  users  engaging? 

(2)  Are  the  right  users  engaging 
with  the  right  PMO  reps? 

(3)  Are  the  right  engagement 
approaches  applied? 

(4)  Are  the  right  events  and  issues 
driving  the  need  to  engage? 

(5)  Are  user  engagement  feedback 
loops  closed  effectively  and  in  a 
timely  manner? 

(6)  Is  the  user  engagement  process 
enabled  (resources,  championship)? 


Engagement  Program 


User  Involvement  Risk  Reduction 
Functionality  Risk  Reduction 
Data/Information  Verification 


Goals,  Impact  &  Value 


Expectations 


Commitments  & 
Relationship  Mgt 


Engagement  & 
Communication 
Methods 


Processes  &  Plans 
Alignment  and  Tracking 
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Proposed  User  Engagement  Metrics  Categories 

Proposed  metrics  categories  fall  into  three  key  measurement 
areas:  enablers,  execution  health,  and  feedback 


Resource  Commitments 

Organizational  Support 

External  Touch  point  Alignment 

Enablers 

" . 


Maturity  of  Overall  User  Engagement  Process 


■ 

■ 5 

I 


. .  _ 


Requirements  Collection 


Engagement  Approaches 


User  Population  Representation  User  Engagement  Levels 


User  Attendance 


_ 


Execution  Health 


Delivery  Promise  Time 


User  Engagement  Impact 


User  Engagement  Feedback:  Closing  the  Loop 


User  Engagement  Effectiveness 


m 


Post  Deployment  Disruption 


Feedback 
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Proposed  High  Priority  User  Engagement  Metrics 


•  The  most  meaningful  metrics  for  assessing  current  user  engagement  progress  and 
impact  will  vary,  but  we  suggest  that  Program  Offices  consider  the  following  process- 
,  financial-,  activity-,  and  outcome-oriented  metrics 


Command  Si  Control  Center 
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About  Metrics  Derivation 

Not  everything  that  can  be  measured  necessarily  should  be 
measured 

It  is  easy  to  identify  metrics;  it  is  much  harder  to  identify  the 
value  of  those  metrics  in  demonstrating  improvement 
progress  and  impact 

Context  must  be  provided  for  metrics  recommendations: 


o  Why  this  metric? 

o  Method  to  measure  and  units  of  measure 
o  Interdependencies  and  strength  of  interdependencies 
o  Importance  of  metric  to  characterization  of  outcome  achievement 
o  Level  of  confidence  that  metric  effectively  communicates  progress 
toward  achievement  of  outcomes 
o  Key  perspectives  of  health  characterized  by  the  metric 

©201 1  The  MITRE  Corporation.  All  rights  reserved 
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Applying  Metrics  Derivation  Lessons 


For  example. 


%  of  engagements  where  appropriate  users 
and  PMO  representatives  are  present 


o 


o 


o 


Why  this  metric? 

x  Our  investigations-to-date  strongly  suggest  that  key  outcomes  associated  with  acceptance  of 
requirements  and  adherence  with  delivery  schedules  are  strongly  influenced  by  the  %  of 
engagements  where  the  right  users  and  PMO  reps  are  present 

Method  to  measure  and  units  of  measure 


( 


#  of  engagements  during  the  specified  timeframe  in  which  the  most  appropriate  users  and  PMO  reps  are  present 


)  X  100 


O 


O 


O 


#  of  user  engagements  during  the  specified  timeframe 

Interdependencies  and  strength  of  interdependencies 

On  a  scale  of  weak  to  strong  influence,  this  metric  is  strongly  influenced  by  "maturity  level  of  user 
engagement  process" 

Moderately  influenced  by  "%  events  conducted  in  user  environment" 

Importance  of  metric  to  characterization  of  outcome  achievement 

On  a  scale  of  slightly  to  very  important,  this  metric  is  moderately  important  to  achievement  of  key 
outcomes  associated  with  acceptance  of  requirements  and  adherence  with  delivery  schedules 

Level  of  confidence  that  metric  effectively  communicates  progress  toward  achievement  of 
outcomes 

On  a  scale  of  1  to  100,  with  100  meaning  extremely  confident,  we  are  75%  confident  that  this 
metric  communicates  progress  toward  achievement  of  outcomes 

Key  perspectives  of  health  characterized  by  the  metric 

Key  perspectives  of  health  characterized  by  this  metric  include  effectiveness  of  user  engagements 
and  efficiency  associated  with  obtaining  user  feedback 
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Relationships  among  Section  804  Principles 


We  then  explored  the  relationships  between  Early  and 
Continual  User  Involvement  and  the  other  three  NDAA  Section 
804  Principles: 


Early  and  Continual 
Involvement  of  the  User 


Prototypes  demonstrate 
feasibility  to  users  and 
provide  a  tangible  means 
to  get  more  definition  on 
users  needs 


V" 


Early,  Successive 
Prototyping  to  Support  an 
Evolutionary  Approach 


MOSA  makes  the 
system  more  flexible 
and  thus  easierto 
accommodate 
changed  user  needs 


■t 


MOSA  enables  insertion  of 
and  demonstration  of 
prototypes  in  an 
operationally  relevant 
environment 


User  provides  priorities  for 
increments.  Increments 
provide  opportunity  for 
frequent  feedback. 


Prototypes  provide  a 

■risk reduction  - > 

method  for  defining 
increment  content. 
Prototypes  can  be 
developed 
incrementally  to 
become  the  final 
delivery _  * 


Multiple,  Rapidly 
Executed  Increments  or 
Releases  of  Capability 


A  stable  MQSA-based 

environment  is  the 
foundation  for  an 
incremental  delivery  culture 


A  Modular,  Open  Systems 
Approach  (MOSA) 


MITRE 


©201 1  The  MITRE  Corporation.  All  rights  reserved 


Command  Si  Control  Center 


Best  Practices  Mapped  to  IT  Acquisition  (perDSB  Report) 


Best  Practices  Mapped  to  New  IT  Acquisition 
Lifecycle  Phases 

Business  Case  Analysis  and 
Development 

Architectural  Development 
and  Risk  Reduction 

Development  & 
Demonstration 

Operations  &  Support 

Early  and  Continual  Involvement  of  the  User 

Voice  of  the  customer 

/ 

/ 

/ 

/  m 

Customer  relationship  management  supported  by 
customer  communications  management 

/ 

/ 

/ 

/ 

Customer  satisfaction  enabled  by  enterprise 
feedback  management 

/ 

/ 

/ 

/ 

Collaboration  management 

/ 

/ 

/ 

User-centered  design  &  Usability 

/ 

Customer  service 

/ 

Multiple,  Rapidly  Executed  Increments  or 
Releases  of  Capabiltiy 

Capability  Maturity  Model  Index  (CMMI)  - 
Acquisition  (AQ) 

/ 

/ 

/ 

/ 

CMMI- Development 

/ 

Incremental  iterative  development  (planning  & 
execution) 

/ 

/ 

/ 

Early,  Successive  Prototyping  to  Support  an 
Evolutionary  Approach 

Demonstration  of  applicable  technology 

/ 

/ 

Demonstration  of  design  possibilities 

/ 

Demonstration  of  requirements  fulfillment 

/  A 

/ 

MOSA 

Establish  Enabling  Environment 

/ 

/ 1 

/ 

Employ  Modular  Design 

/ 

/ 

Designate  Key  Interfaces 

/ 

/ 

Use  Open  Standards 

/  m 

/ 

Certify  Conformance 

/ 
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Key  Discoveries 


•  Government  program  application  of  some  DSB-recommended  principles 

(e.g.,  Multiple ,  Rapidly  Executed  Increments  or  Releases  of  Capability)  is  more 
advanced  than  for  other  principles  (e.g..  Early  and  Continual  Involvement  of 
the  User) 

•  Considerable  performance  data  is  typically  collected;  should  investigate  its 
effectiveness  for  IT  Acquisition  programs 

•  Standardized  methods  within  the  DoD  for  selecting  acquisition  program 
metrics  and  monitoring  performance  could  not  be  identified 

•  Measuring  adoption  of  the  principles  will  require  considering  program 
circumstances 

•  The  four  DSB-recommended  principles  within  NDAA  Section  804  are  not 
necessarily  the  only  important  principles 

•  Need  to  share  a  common  understanding  of  how  the  acquisition  principles 
link  to  desired  outcomes 

©  201 1  The  MITRE  Corporation.  All  rights  reserved  “j  /. . 
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Recommendations  for  Your  Program  Office 


•  Provide  additional  venues  for  users  to  communicate  with 
procurement  professionals  (acquirers  and  developers) 

•  Let  users  know  where  their  system  program  office  is  and  how 
to  provide  good  ideas  to  them 

•  Plan  to  align  in  situ  capability  development  sessions  with 
program  increment  planning  to  reduce  requirements  ambiguity 

•  When  many  systems  deploy  to  a  location,  conduct  a  system 
environment  study  to  determine  impacts  on  user  productivity 

•  Formulate  an  alliance  with  operating  agencies  to  help  alleviate 
non-performance  of  systems  when  deployed 
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Applying  What  We  Learned  to 
Developing  A  New  System 


USING  HIGH  CONTACT  USER  ENGAGEMENT  METHODS,  SUCH 
AS  GAMING,  TO  DEVELOP  ACQUISITION  STRATEGIES  FOR 
COMPOSABLE  CAPABILITIES  ON-DEMAND  (CCOD®) 
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What  is  Composable  Capability  on  Demand  (CCOD  ®) 


CCOD* 

Composable  Capability  on  Demand 


Enabling  Great  Composers 
of  the  21st  Century 


•  A  set  of  technical  abilities  that  will  enable  DoD 
and  civilian  users  to  dynamically  assemble  and 
employ  elements  of  the  C4ISR  enterprise 

•  Will  allow  the  non-technocenti  to  adapt  their 
enterprise  according  to  the  nature  and  scale  of 
the  mission 

•  Not  a  system 

o  CCOD  consists  of  resources  that  can  be  formed  or 
re-formed  as  needed 

*  These  resources  are  embedded  within  a  distributed 
hybrid  (fixed  and  mobile)  infrastructure 
environment,  that  may  not  be  locally  provisioned 

•  Draws  mission  information  from  traditional  and 
nontraditional  data  sources  to  enhance  situation 
awareness,  collaboration,  social  networking,  and 
decision  support 

•  Will  rely  on  a  composable  computational  and 
network  infrastructure  for  mission  assurance 


MITRE 


©201 1  The  MITRE  Corporation.  All  rights  reserved 


17 


Acquisition  for  Composable  Systems 


•  Goal:  Engage  with  various  users  of  a  proposed  process 
to  acquire  and  sustain  composable  systems 

•  Activities  [*] 

o  Proposing  acquisition  approaches  to  achieve  CCOD®  objectives 

o  Constructing  games  highlighting  particular  aspects  of  proposed  CCOD® 
acquisition,  and  conducting  exercises  with  Subject  Matter  Experts  (SMEs) 
so  that  we  can  assess  the  value  of  the  games  for  learning  and  evaluating 
acquisition  effectiveness 

o  Creating  an  environment  to  enable  CCOD®  acquisition  game  play: 
x  Tabletop  exercises 

Electronic  gaming  in  a  distributed  and  asynchronous  fashion 

o  The  environment  may  then  be  extended  to  experiment  with  a  wide  variety 
of  acquisition  processes  with  participation  from  many  different 
stakeholders 


l*1  From  MITRE  Public  Release  Approval  Case:  11-1622 
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